Apparatus for ai-based automatic ultrasound diagnosis of liver steatosis and remote medical diagnosis method using the same

ABSTRACT

Disclosed herein are an apparatus for AI-based automatic ultrasound diagnosis of liver steatosis and a remote medical diagnosis method using the same applied in the field of ultrasound image processing. The apparatus for AI-based automatic ultrasound diagnosis of liver steatosis can automatically determine a grade of liver steatosis, which is difficult to determine visually, through extraction from an image acquired by imaging medical examination using a deep learning trained artificial neural network.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of and priority to Korean PatentApplication 10-2019-0059049, filed May 20, 2019, the entire disclosureof which is incorporated herein by reference.

FIELD

The present invention relates to an apparatus for AI-based automaticultrasound diagnosis of liver steatosis which can automaticallydetermine a grade of liver steatosis, which is difficult to determinevisually, through extraction from an image acquired by imaging medicalexamination using a deep learning-trained artificial neural network, anda remote medical diagnosis method using the same. More particularly, thepresent invention relates to an apparatus for AI-based automaticultrasound diagnosis of liver steatosis which detects a liver region anda kidney region required for diagnosis of liver steatosis from anultrasound image, concatenates and combines images of the two regionsinto one integrated image, decomposes the integrated image into foursub-band images, that is, an LL image, an LH image, an HL image, and anHH image, by discrete wavelet transform, trains independent artificialneural networks on the respective sub-band images, and automaticallydetermines a grade of liver steatosis in the patient based on theultrasound image of the patient received from an ultrasound probe sensorusing the deep learning-trained artificial neural networks, and a remotemedical diagnosis method using the same.

In addition, the present invention is advantageously used inimplementing a virtual doctor that automatically analyzes the degree ofliver steatosis in a patient using a deep learning-trained AI platform,notifies the patient or a doctor of an analysis result, and provides aremote consultation service via the Internet.

BACKGROUND

This application is a continuation of an earlier-issued European patenttitled “Remote Medical-Diagnosis System and Method” (issued on Dec. 12,2018, Patent Ser. No. 02/140,412).

The earlier-issued patent discloses a remote medical diagnosis systemand a remote medical diagnosis method using the same, the remote medicaldiagnosis system including: a bioanalyzer including a bio-disc or alab-on-a-disc adapted to receive a sample therein to perform abiological, chemical or biochemical reaction; a virtual doctor includinga medical examination device including a thermometer, asphygmomanometer, a camera, a stethoscope, a body fat analyzer, avascular screening device, an ultrasound imager, a urinalysis device, apulsimeter, a blood collection device, an electrocardiographer, an X-raydevice, an oxygen saturation tester, a dementia testing device, acomputerized axial tomographer (CAT), a magnetic resonance imager (MRI),a capsule endoscope, a magnifier, a camera-integrated magnifier, and abioshirt having a function of measuring biological signals (diabetes,obesity, blood pressure, pulse, electrocardiogram, body temperature, andthe like), the virtual doctor residing as software in a user terminal toguide or instruct how to use the bioanalyzer and the medical examinationdevice and provide a consultation service with a user; a user terminalproviding a consultation service with a medical expert or the virtualdoctor; a medical expert terminal providing a consultation service witha user; and a remote diagnosis server connecting a user to a medicalexpert as a consultation specialist during a regular medical check-upperiod, connecting a user to the virtual doctor during the otherperiods, and blocking connection between the user and the virtual doctorif the regular medical check-up period elapses without consultation withthe medical expert.

Recently, as digital image processing technology has been used in thefield of clinical diagnosis along with medical device manufacturingtechnology, there have been many advances in diagnostic radiology.

In particular, ultrasound diagnosis is thus harmless to the human bodyby allowing avoidance of exposure to harmful radiation, as compared withCT or X-ray medical equipment, allows acquisition of a cross-sectionalimage of the human body in a non-invasive manner, and is portable andinexpensive. Particularly, ultrasound diagnosis allows real-time imageacquisition and thus real-time observation of movement of an organ.

Such an ultrasound diagnostic technique is widely used to determine thedegree of liver steatosis using the fact that reflection properties ofultrasound waves significantly differ between water and fat.

Fatty liver is the most common disease and is detected by abdominalultrasonography. Recently, diagnosis of liver steatosis is mainlycarried out by measuring the brightness level or texture properties ofan abdominal cross-sectional image obtained by ultrasound equipment andcalculating a hepatorenal sonographic index (HI), which is a measure fordetermining the degree of steatosis in liver tissue, wherein the HI isgenerally a ratio of mean brightness level or degree of texture betweena liver and a right kidney on an echogenicity histogram of a cortex.However, diagnosis of fatty liver based on the hepatorenal sonographicindex (HI) obtained using the brightness level or the texturecharacteristics of an ultrasound image has a problem in that calculationerrors are likely to occur due to low resolution of the ultrasound imageand severe noise. In addition, the ultrasound image is often severelydamaged, causing difficulty in accurate medical interpretation bynonprofessionals. Further, fatty liver is divided into four grades:normal, mild, moderate, and severe, and, in more severe cases, may bediagnosed as liver cirrhosis or liver cancer.

Inaccuracy of calculation of the HI leads to increase in ambiguity ofdetermination of the fatty liver grade using ultrasound examination and,eventually, the determination depends on subjective judgment of anexaminer, causing inconsistency between opinions of different examinersand confusion in reading results of ultrasound examination.

The present invention has been conceived to solve such problems in theart and provides an apparatus for AI-based automatic ultrasounddiagnosis of liver steatosis which extracts images of a liver region anda kidney region, concatenates and combines the images of the two regionsinto one integrated image, performs discrete wavelet transform on theintegrated image to acquire sub-band images, trains artificial neuralnetworks on the respective sub-band images, and automatically determinesa grade of liver steatosis in a patient using the deep learning-trainedartificial neural networks, and a remote medical diagnosis method usingthe same.

RELATED LITERATURE Patent Document

-   (Patent document 0001) European Patent No. 02140412 (issued on Dec.    12, 2018)

Non-Patent Document

-   (Non-patent document 0001) (Article 1) Michal Byra and Grzegorz    Styczynski, et al, “Transfer learning with deep convolutional neural    network for liver steatosis assessment in ultrasound images”,    International Journal of Computer Assisted Radiology and Surgery,    August, 2018-   (Non-patent Document 0002) (Article 2) Marshall R H, Eissa M, Bluth    E I, Gulotta P M, Davis N K “Hepatorenal index as an accurate,    simple, and effective tool in screening for steatosis” American    Journal of J Roentgenology Vol. 199 (2012), pp. 997-1002-   (Non-patent Document 0003) (Article 3) Christian Szegedy et al.,    Going Deeper with Convolutions, 2015 Computer Vision and Pattern    Recognition-   (Non-patent Document 0004) (Article 4) Christian Szegedy et al.,    Rethinking the Inception Architecture for Computer Vision, 2016    Computer Vision and Pattern Recognition-   (Non-patent Document 0005) (Article 5) Liang-Chieh Chen et al.,    Rethinking Atrous Convolution for Semantic Image Segmentation, 2017    Computer Vision and Pattern Recognition

SUMMARY

The present invention has been conceived to solve such problems in theart and it is one aspect of the present invention to provide anapparatus for AI-based automatic ultrasound diagnosis of liver steatosiswhich can automatically screen liver steatosis based on an ultrasoundimage of a patient using artificial intelligence (AI), the apparatus forAI-based automatic ultrasound diagnosis of liver steatosis detecting aliver region and a kidney region required for diagnosis of liversteatosis from the ultrasound image, concatenating and combining imagesof the two regions into one integrated image, decomposing the integratedimage into four sub-band images, that is, an LL image, an LH image, anHL image, and an HH image by discrete wavelet transform, trainingindependent artificial neural networks on the respective sub-bandimages, and automatically determining a grade of liver steatosis in thepatient based on the ultrasound image of the patient received from anultrasound image sensor using the deep learning-trained artificialneural networks.

It is another aspect of the present invention to provide a remotemedical diagnosis method which automatically analyzes liver steatosis ina patient using the apparatus for AI-based automatic ultrasounddiagnosis of liver steatosis set forth above, notifies the patient or adoctor of an analysis result via the Internet, and provides aconsultation service with a medical expert using a virtual doctorfurther provided.

However, it should be understood that the technical problem to be solvedby embodiments of the present invention is not limited to theaforementioned technical problems and other technical problems mayexist.

In accordance with one aspect of the present invention, an apparatus forAI-based automatic ultrasound diagnosis of liver steatosis includes: anultrasound probe sensor acquiring an ultrasound image from a patient; aregion-of-interest extraction unit acquiring images of one or moreregions of interest helpful to diagnosis of liver steatosis from theultrasound image; an image integration unit concatenating the images ofthe one or more regions of interest into one integrated image; adarkness-to-shape transform kernel transforming the integrated imageinto a shape-based integrated image; and an artificial neural networktrained on the shape-based integrated image by deep learning, whereinthe deep learning-trained artificial neural network automaticallydetermines a grade of liver steatosis in the patient based on theultrasound image of the patient received from the ultrasound probesensor.

In accordance with another aspect of the present invention, an apparatusfor AI-based automatic ultrasound diagnosis of liver steatosis includes:an ultrasound probe sensor acquiring an ultrasound image from a patient;a region-of-interest extraction unit acquiring images of one or moreregions of interest helpful to diagnosis of liver steatosis from theultrasound image; an image integration unit including a rearrangementunit rearranging image pixels of each of the images of the regions ofinterest acquired by the region-of-interest extraction unit to obtain apattern image, the image integration unit concatenating the patternimages into one integrated image; and an artificial neural networktrained on the integrated image by deep learning, wherein the deeplearning-trained artificial neural network automatically determinesliver steatosis in the patient based on the ultrasound image of thepatient received from the ultrasound probe sensor.

The artificial neural network according to the present invention may bea convolutional neural network (CNN) or a recurrent neural network(RNN).

In the present invention, the artificial neural network is a neuralnetwork that can be trained by deep learning, and may include at leastone layer or element selected from the group of a convolution layer, apooling layer, an ReLu layer, a transpose convolution layer, anunpooling layer, a 1×1 convolutional layer, a skip connection, a globalaverage pooling (GAP) layer, a fully connected layer, a long short termmemory (LSTM), a softmax classifier, an auxiliary classifier, and asupport vector machine (SVM). For example, the artificial neural networkmay be an artificial neural network which further includes an operationunit for batch normalization upstream of the ReLu layer.

The region-of-interest extraction unit may set a region of interest forultrasound diagnosis for a patient on the ultrasound image using arectangular or elliptical window to extract an image sample within aregion of the window.

Accordingly, only darkness intensity of pixels within the window differsbetween the extracted images of the regions of interest depending on thedegree of liver steatosis in a patient, and the extracted images allhave a rectangular or elliptical shape, which corresponds to the shapeof the window.

That is, although fatty liver is classified into four grades, that is,normal, mild, moderate, and severe, based on the degree of steatosis,the extracted images of the regions of interest have the same outershape regardless of the grade of fatty liver. In other words, only thedarkness intensity of pixels within the window differs between theextracted images depending on the grade of fatty liver, and outer shapesof the extracted images are identical to the shape of the window used.

In addition, different integrated images, which are acquired bycombining the images of several regions of interest into one, have auniform outer shape. That is, since integrated images indicative ofdifferent fatty liver grades have the same outer shape, it is impossibleto determine the grade of fatty liver based on the outer shape of theintegrated image. In other words, integrated images indicative ofdifferent fatty liver grades differ only in darkness intensity of pixelstherein.

Unfortunately, known artificial neural networks are designed to learnand recognize an outer shape of an image, rather than darkness intensityof image pixels.

For example, the reason why existing artificial neural networksdistinguish a car from a cat is that the two objects have differentouter shapes. Since the extracted integrated images have the same outershape despite being indicative of different fatty liver grades, it isvery difficult for existing artificial neural networks to efficientlylearn and recognize the images.

In order to solve this problem, the apparatus according to the presentinvention includes the darkness-to-shape transform kernel transformingthe darkness intensity-based image into a shape-based image. Thedarkness-to-shape transform kernel transforms a lower-dimensional imageinto a higher-dimensional image to generate an image having an outershape that varies depending on data of darkness intensity of pixels. Inthe present invention, an image obtained through transformation by thedarkness-to-shape transform kernel depending on data of darknessintensity of pixels in an image obtained by the region-of-interestextraction unit is referred to as “shape-based image”.

In addition, an image upstream of the darkness-to-shape transform kernelis referred to as “darkness intensity-based image”.

The darkness-to-shape transform kernel transforms the darknessintensity-based image into the shape-based image.

For example, suppose that an image extracted from a first patient by theregion-of-interest extraction unit is indicative of normal liversteatosis and an image extracted from a second patient by theregion-of-interest extraction unit is indicative of severe liversteatosis. Since all regions of interest in the ultrasound images wereextracted using a predetermined window at the time of extraction, theextracted images have the same outer shape, which corresponds to theshape of the window. Therefore, it is not only difficult to distinguishbetween liver steatosis grades of the two patients based on the outershapes of the extracted images, but it is also difficult for anartificial neural network to efficiently learn and recognize theextracted images.

However, since the distribution and pattern of darkness intensity ofpixels significantly differ between the darkness intensity-based imagesextracted from the two patients, shape-based images having differentouter shapes can be obtained through transformation of the darknessintensity-based images by the darkness-to-shape transform kernel. Sincethe shape-based images have different outer shapes, the artificialneural network can efficiently learn and recognize the shape-basedimages.

The darkness-to-shape transform kernel may be any one selected from thegroup of a linear kernel, a polynomial kernel, a Gaussian kernel, ahyperbolic tangent kernel, and a radial basis function (RBF) kernel,which are well known to those skilled in the art.

Preferably, the darkness-to-shape transform kernel is a polynomialkernel given by the following equation:

(x ₁ ,x ₂)→(x ₁ ²,√{square root over (2)}x ₁ x ₂ ,x ₂ ²)

On the left side of the equation, x₁ may denote an address of each pixelin a darkness intensity-based image and x₂ may denote darkness intensityof the pixel.

The darkness-to-shape transform kernel transforms the darknessintensity-based image given by (x₁,x₂) in the left side of the equationinto a shape-based image expressed by (x₁ ², √{square root over(2)}x₁x₂,x₂ ²) in the right side of the equation.

Here, the shape-based image is displayed in a three-dimensional spaceand is expressed by darkness intensity at a location in atwo-dimensional plane. For example, x₁ ² may denote a coordinate valueon the horizontal axis in the two-dimensional plane, √{square root over(2)}x₁x₂ may denote a coordinate value on the vertical axis in thetwo-dimensional plane, and x₂ ² may denote darkness intensity of a pixelat a location represented by the coordinate value on the horizontal axisand the coordinate value on the vertical axis.

In another embodiment, the shape-based image may be displayed in athree-dimensional space and may be represented by darkness intensity ina two-dimensional plane, wherein x₁ ² may denote a coordinate value onthe horizontal axis in the two-dimensional plane, x₂ ² may denote acoordinate value on the vertical axis in the two-dimensional plane, and√{square root over (2)}x₁x₂ may denote darkness intensity of a pixel ata location represented by the coordinate value on the horizontal axisand the coordinate value on the vertical axis.

In addition, x₁, which denotes the address of each pixel in the darknessintensity-based image, may be a serial number assigned to the pixel.

For example, if the region-of-interest extraction unit uses arectangular window to extract a region of interest from an ultrasoundimage and the size of the window is set to 3×3, x₁ is a number between 1and 9 since the total number of pixels is 9.

In addition, if an ultrasound image having 256 gray levels is used, x₂,which represents darkness intensity of each pixel in a region ofinterest of the ultrasound image, is a number between 0 and 255.

In another embodiment, x₁ may be the sum of the x-axis and y-axiscoordinate values of each pixel.

The image integration unit according to the present invention mayfurther include a rearrangement unit that receives a first image, asecond image, a kidney border image, and a third image from theregion-of-interest extraction unit, the first image being obtained byextracting a region including a kidney and a liver from the ultrasoundimage, the second image being obtained by extracting only a kidneyregion from the first image, the kidney border image being obtained byextraction along a boundary between the first image and the secondimage, and the third image being obtained by extracting the remainingportion of the first image excluding the second image and the kidneyborder image; generates a pattern image of the second image by takingimage pixels from the second image while circularly moving starting fromcenter coordinates of the second image and rearranging the taken imagepixels into a rectangular image; generates a pattern image of the kidneyborder image taking image pixels from the kidney border image whilecircularly moving along the kidney border image and rearranging thetaken image pixels into a rectangular image; and generates a patternimage of the third image by taking image pixels from the third imageaccording to a predetermined pixel scanning scheme and rearranging thetaken images into a rectangular image. The image integration unitconcatenates the generated pattern images into one integrated image.

Since image patterns in the pattern image of the second image, thepattern image of the kidney border image, and the pattern image of thethird image significantly differ between different liver steatosisgrades, the artificial neural network can efficiently learn andrecognize the integrated image.

In one embodiment, the pixel scanning scheme may be set such that therearrangement unit takes image pixels from the third image while movingfrom left to right and from top to bottom of the third image.

In another embodiment, the pixel scanning scheme may be set such thatthe rearrangement unit takes image pixels from the third image whilecircularly rotating in a direction of increasing radius from an insideof the third image.

In accordance with a further aspect of the present invention, anapparatus for AI-based automatic ultrasound diagnosis of liver steatosisincludes: an ultrasound probe sensor acquiring an ultrasound image froma patient; a region-of-interest extraction unit obtaining images of oneor more regions of interest from the ultrasound image; an imageintegration unit combining the images of the one or more regions ofinterest into one integrated image; a wavelet transform unit decomposingthe integrated image into sub-band images by discrete wavelet transform;and an artificial neural network trained on the sub-band images by deeplearning, wherein the deep learning-trained artificial neural networkautomatically determines a grade of liver steatosis in the patient basedon the ultrasound image of the patient received from the ultrasoundprobe sensor.

In another embodiment, the apparatus for AI-based automatic ultrasounddiagnosis of liver steatosis may further include a darkness-to-shapetransform kernel transforming the sub-band images into one shape-basedsub-band image, wherein the artificial neural network may be trained onthe shape-based sub-band image by deep learning.

The region-of-interest extraction unit may extract at least two selectedfrom the group of an image of a liver parenchyma region, an image of akidney parenchyma region, an image of a right portal vein (RPV) region,an image of a hepatic vein region, an image of a kidney region, an imageof a spleen region, and an image of a diaphragm region from theultrasound image.

In the present invention, the discrete wavelet transform unit maydecompose a two-dimensional ultrasound image into four sub-band images,that is, an LL image, an LH image, an HL image, and an HH image, byapplying a low pass filter and a high pass filter in a horizontal orvertical direction of a two-dimensional ultrasound image, followed bydownsampling by a factor of 2.

Here, the sub-band LL image is acquired by applying the low pass filterin both horizontal and vertical directions of an original ultrasoundimage, followed by subsampling by a factor of ½ in each of thehorizontal and vertical directions.

The sub-band HL image is acquired by applying the low pass filter andthe high pass filter in the horizontal and vertical directions of theoriginal ultrasound image, respectively, followed by subsampling by afactor of ½ in each of the horizontal and vertical directions.

The sub-band LH image is acquired by applying the high pass filter andthe low pass filter in the horizontal and vertical directions of theoriginal ultrasound image, respectively, followed by subsampling by afactor of ½ in each of the horizontal and vertical directions.

The sub-band HH image is acquired by applying the high pass filter inboth the horizontal and vertical directions of the original ultrasoundimage, followed by subsampling by a factor of ½ in each of thehorizontal and vertical directions.

In another embodiment, the darkness-to-shape transform kernel unit maytransform each of the four sub-band images into a shape-based sub-bandimage to generate a shape-based sub-band LL image, a shape-basedsub-band LH image, a shape-based sub-band HL image, and a shape-basedsub-band HH image, and the artificial neural network may include a firstartificial neural network trained on the shape-based sub-band LL image,a second artificial neural network trained on the shape-based sub-bandLH image, a third artificial neural network trained on the shape-basedsub-band HL image, and a fourth artificial neural network trained on theshape-based sub-band HH image.

In the present invention, the organ of interest may include a liver, aright portal vein (RPV), a hepatic vein, a kidney, a spleen, and adiaphragm, and the ultrasound image may be an ultrasound image obtainedby placing the ultrasound probe sensor in a parasagittal scan plane withrespect to an affected area. Preferably, the organ of interest includesa liver and a kidney.

In another embodiment, the region-of-interest extraction unit mayinclude an artificial neural network performing semantic segmentation onthe ultrasound image to acquire a semantic segmented ultrasound image inwhich different organs of interest are labeled with different values ordifferent colors for extraction of the images of the regions ofinterest. The region-of-interest extraction unit may generate a firstimage by extracting a region including the kidney and the liver amongthe organs of interest from the ultrasound image by semanticsegmentation, generate a second image by extracting only a kidney regionfrom the first image by semantic segmentation, generate a kidney borderimage by extraction along a boundary between the first image and thesecond image, generate a third image by extracting the remaining portionof the first image excluding the second image and the kidney borderimage, and transmit the second image, the kidney border image, and thethird image to the image integration unit.

Here, the rearrangement unit of the image integration unit may generatea pattern image of the second image by taking image pixels from thesecond image while circularly moving starting from center coordinates ofthe second image and rearranging the taken image pixels into arectangular image; generate a pattern image of the kidney border imagetaking image pixels from the kidney border image while circularly movingalong the kidney border image and rearranging the taken image pixelsinto a rectangular image; and generate a pattern image of the thirdimage by taking image pixels from the third image according to apredetermined pixel scanning scheme and rearranging the taken imagesinto a rectangular image. The image integration unit concatenates thegenerated pattern images into one integrated image.

Semantic segmentation may be performed using an artificial neuralnetwork that detects a location of an organ corresponding to a specificclass in a given ultrasound image by object classification on apixel-by-pixel basis and separates the organ from the other organs,provided that the organ is present in the ultrasound image.

The semantic segmented ultrasound image may be a color map of organswhich is configured by preassigning different colors to differentorgans.

For example, the color map may be configured by assigning green to aliver, yellow to a kidney, blue to a spleen, and orange to a diaphragm,Here, semantic segmentation may be performed based on color.

In a further embodiment, the region-of-interest extraction unit mayinclude: a feature point detection unit detecting a feature point in theultrasound image; a reference feature point storage unit storing afeature point of a reference organ image; and an organ matching unitfinding an organ of interest from the ultrasound image through an organmatching process that compares similarity between the feature pointdetected by the feature point detection unit and the feature point ofthe reference organ image stored in the reference feature point storageunit. The region-of-interest extraction unit according to thisembodiment may generate a first image by extracting a region includingthe kidney and the liver among the organs of interest from theultrasound image, generate a second image by extracting only a kidneyregion from the first image, generate a kidney border image byextraction along a boundary between the first image and the secondimage, and generate a third image by extracting the remaining portion ofthe first image excluding the second image and the kidney border image.

The reference organ image may include images of the organs of interestsuch as the liver, the kidney, the spleen, and the diaphragm, which areacquired from an image indicative of normal liver steatosis. In theorgan matching process, locations of the organs of interest in anultrasound image are detected based on the feature point of thereference organ image.

The organ matching process may be performed using only selected pixelssuch as feature points (key points). Specifically, the organ matchingprocess may be performed by finding feature points in two or moreimages, such as a point, line, border, or edge component, and matchingthe feature points to one another. A corner point may be used as thefeature point.

The corner point may include pixels which can be easily identified evenwhen the shape, size, or position of an object changes and locations ofwhich can be easily detected in an image even when illumination of acamera changes. Specifically, the corner point may be a pixel thatcauses significant changes to an image in all directions (in vertical,horizontal and diagonal directions) when a small predetermined windowmovable vertically and laterally is shifted on the image while graduallyscanning the image.

Examples of an algorithm for extracting the corner point or the featurepoint may include Harris corner detection, scale-invariant featuretransform (SIFT), speeded-up robust features (SURF) and features fromaccelerated segment test (FAST), which are well known to those skilledin the art.

In the present invention, a small image acquired by collecting pixelsaround a feature point of an ultrasound image to be matched is referredto as a “patch image”.

In the present invention, the feature point of the reference organ imageis referred to as “reference feature point” and the feature point of thepatch image is referred to as “matching feature point”. The organmatching process includes a process of matching the matching featurepoint to the reference feature point by geometric transformation.Geometric transformation includes translation, rotation, and scaling ina coordinate space.

In the present invention, after comparing similarity between featurepoints of two images (the reference organ image and the patch image) andperforming geometric transformation, a cross-correlation coefficientbetween the two images may be calculated. If the cross-correlationcoefficient is high, it is determined that a corresponding organ islocated in the patch image.

That is, the feature point of the reference organ image is compared withthe feature point of each patch image to select a patch image having afeature point with high similarity to the reference feature point,thereby generating a pair of feature points between the two images (thereference organ image and the selected patch image). Then, acorrespondence between the feature points in the two images isascertained to determine a geometric transformation relation between thetwo images. Then, the patch image is aligned with the reference organimage by geometric transformation based on the determined geometrictransformation relation, followed by calculating a cross-correlationcoefficient between the two images (the reference organ image and thegeometrically transformed patch image). If the cross-correlationcoefficient between the two images is greater than a predeterminedthreshold, it is determined that a corresponding organ is located in acurrent patch image in the ultrasound image.

Each of the feature points may contain data of the coordinate locationand orientation thereof. Using these data, the geometric transformationrelation between the two images (the reference organ image and the patchimage) may be determined based on the pair of feature points between thetwo images.

The orientation data may be calculated based on gradient direction andmagnitude calculated from pixels around the feature point.

The cross-correlation coefficient may be calculated using any oneselected from the group of sum of squared difference (SSD), sum ofabsolute difference (SAD), k-nearest neighbor algorithm (KNN), andnormalized cross correlation (NCC).

In yet another embodiment, the region-of-interest extraction unit maydetect and extract a location of an organ of interest in the ultrasoundimage through wavelet frame or redundant (over-complete) wavelet-basedorgan matching.

In wavelet frame or redundant (over-complete) wavelet-based organmatching, an overlapping pixel determination unit may be used to selectoverlapping pixels between a sub-band HL frame image, a sub-band LHframe image, and a sub-band HH frame image acquired by wavelet frametransform of the ultrasound image as the feature point (key point orcorner point).

The overlapping pixels may be pixels having a value greater than afeature point determination threshold, wherein the pixels are found byperforming pixel-by-pixel multiplication between the sub-band HL frameimage, the sub-band LH frame image, and the sub-band HH frame imageacquired by wavelet frame transform of the ultrasound image, followed byapplication of the feature point determination threshold on apixel-by-pixel basis. The overlapping pixels may be used as the featurepoint of the ultrasound image.

Then, a small image is formed by collecting pixels around the featurepoint in the ultrasound image to acquire the patch image.

Since the sub-band HL frame image has a highlighted horizontal edgecomponent and the sub-band LH frame image has a highlighted verticaledge component, the sub-band HL frame image and the sub-band LH frameimage can sufficiently provide edge feature points that can beconveniently used for organ matching.

In addition, the overlapping pixel determination unit may also be usedto select overlapping pixels between a sub-band HL frame image, asub-band LH frame image, and a sub-band HH frame image acquired bywavelet frame transform of the reference organ image as the featurepoint of the reference organ image (that is, the reference featurepoint), wherein the overlapping pixels may be pixels having a valuegreater than a feature point determination threshold, which are found byperforming pixel-by-pixel multiplication between the sub-band HL frameimage, the sub-band LH frame image, and the sub-band HH frame image,followed by application of the feature point determination threshold ona pixel-by-pixel basis.

That is, the overlapping pixels of the reference organ image obtained bythe overlapping pixel determination unit are selected as the referencefeature point and the overlapping pixels of the ultrasound imageobtained by the overlapping pixel determination unit are used as thematching feature point.

Then, the feature point of the reference organ image is compared withthe feature point of each patch image to select patch images having afeature point with high similarity to the reference feature point,thereby generating a pair of feature points between the two images (thereference organ image and each of the selected patch images). Then, acorrespondence between the feature points in the two images isascertained to determine a geometric transformation relation between thetwo images. Then, the patch image is aligned with the reference organimage by geometric transformation based on the determined geometrictransformation relation, followed by calculating a cross-correlationcoefficient between the two images (the reference organ image and thegeometrically transformed patch image). If the cross-correlationcoefficient between the two images is greater than a predeterminedthreshold, it is determined that a corresponding organ is located in acurrent patch image in the ultrasound image.

In accordance with yet another aspect of the present invention, a remotemedical diagnosis method using the apparatus for AI-based automaticultrasound diagnosis of liver steatosis according to the presentinvention includes the steps of: extracting images of one or moreregions of interest helpful to diagnosis of liver steatosis from anultrasound image of a patient; concatenating and combining the images ofthe one or more regions of interest into one integrated image;transforming the integrated image into a shape-based integrated image ora pattern image; training the artificial neural network on theshape-based integrated image or the pattern image by deep learning;automatically determining, by a virtual doctor, a grade of liversteatosis based on the ultrasound image of the patient; and providing,by a remote medical diagnosis system, a remote consultation service witha medical expert.

It should be understood that the aforementioned solutions are providedfor illustration only and are not to be construed in any way as limitingthe present invention. In addition to the exemplary embodimentsdescribed above, other embodiments may exist in the drawings anddetailed description of the invention.

As described above, in the field of ultrasound image processing, thepresent invention provides an apparatus for AI-based automaticultrasound diagnosis of liver steatosis that can automatically determinethe grade of liver steatosis, which is difficult to determine visually,through extraction from an image acquired by imaging medical examinationusing a deep learning trained artificial neural network, and a remotemedical diagnosis method using the same.

However, it should be understood that the effects obtainable by thepresent invention are not limited to the aforementioned effects andother effects may exist.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an apparatus for AI-based automaticultrasound diagnosis of liver steatosis according to one embodiment ofthe present invention.

FIGS. 2A and 2B are a block diagram of an apparatus for AI-basedautomatic ultrasound diagnosis of liver steatosis according to anotherembodiment of the present invention embodiment, wherein adarkness-to-shape transform kernel transforming sub-band images into ashape-based sub-band image is further disposed in-line between a wavelettransform unit and an artificial neural network to train the artificialneural network on the shape-based sub-band images by deep leaning.

FIG. 3 is a diagram illustrating a process of acquiring a pattern imageand an integrated image by an image integration unit.

FIG. 4 is a block diagram of a region-of-interest extraction unitextracting a region of interest from an ultrasound image acquired by anultrasound probe sensor according to one embodiment of the presentinvention.

FIG. 5 shows an exemplary ultrasound image in which different organs arelabeled with different colors, the ultrasound image being obtained bysemantic segmentation of an ultrasound image in a parasagittal scanplane.

FIG. 6 is a diagram of a feature point detection unit selectingoverlapping pixels between sub-band frame images as a feature pointaccording to one embodiment of the present invention.

FIG. 7 shows exemplary sub-band frame images obtained by applyingwavelet frame transformation to an ultrasound image.

FIG. 8 is a diagram of an apparatus for AI-based automatic ultrasounddiagnosis of liver steatosis provided with a remote diagnosis systemaccording to one embodiment of the present invention.

DETAILED DESCRIPTION

Now, exemplary embodiments of the present invention will be described indetail with reference to the accompanying drawings so that those skilledin the art can easily implement the present invention. It should beunderstood that the present invention is not limited to the followingembodiments and may be embodied in different ways. In the drawings,portions irrelevant to the description will be omitted for clarity. Likecomponents will be denoted by like reference numerals throughout thespecification.

It will be understood that when an element is referred to as being“connected to” another element, it can be directly connected to theother element, or can be electrically or indirectly connected to theother element with a different element interposed therebetween.

It will be understood that when an element is referred to as being “on,”“above,” “at an upper end of,” “under,” “below,” “at a lower end of”another element, it may directly adjoin the other element or layer, orintervening elements may be present.

It will be further understood that the terms “comprises” and/or“comprising,” when used in this specification, specify the presence ofstated features, steps, operations, elements, and/or components, but donot preclude the presence or addition of one or more other features,steps, operations, elements, components, and/or groups.

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram of an apparatus for AI-based automaticultrasound diagnosis of liver steatosis 600 according to one embodimentof the present invention. The apparatus for AI-based automaticultrasound diagnosis of liver steatosis according to this embodimentincludes: an ultrasound probe sensor 30 acquiring an ultrasound imagefrom a patient; a region-of-interest extraction unit 32 extractingimages of one or more regions of interest from the ultrasound image; animage integration unit 34 concatenating the images of the one or moreregions of interest into one integrated image; a darkness-to-shapetransform kernel 36 transforming the integrated image into a shape-basedintegrated image; and an artificial neural network 38 trained on theshape-based integrated image by deep learning, wherein the deeplearning-trained artificial neural network 38 automatically determines agrade of liver steatosis in the patient as normal, mild, moderate,severe, or cirrhosis based on the ultrasound image of the patientreceived from the ultrasound probe sensor 30.

Referring to FIG. 1, in another embodiment, the image integration unit34 may further include a rearrangement unit 34 a to acquire a patternimage of each of the images of the regions of interest. The imageintegration unit 34 may concatenate the pattern images into oneintegrated image and the artificial neural network 38 may be directlytrained on the integrated image by deep learning. In this case, thedarkness-to-shape transform kernel 36 may be omitted.

FIG. 2 is a block diagram of an apparatus for AI-based automaticultrasound diagnosis of liver steatosis 600 according to anotherembodiment of the present invention. The apparatus for AI-basedautomatic ultrasound diagnosis of liver steatosis according to thisembodiment includes: an ultrasound probe sensor 30 acquiring anultrasound image from a patient; a region-of-interest extraction unit 32extracting images of one or more regions of interest from the ultrasoundimage; an image integration unit 34 concatenating the images of the oneor more regions of interest into one integrated image; a wavelettransform unit 35 decomposing the integrated image into sub-band imagesby discrete wavelet transform; and an artificial neural network trainedon the sub-band images by deep learning, wherein the deep learningtrained artificial neural network automatically determines a grade ofliver steatosis in the patient based on the ultrasound image of thepatient received from the ultrasound probe sensor 30.

Referring to FIG. 2, the apparatus for AI-based automatic ultrasounddiagnosis of liver steatosis according to this embodiment may furtherinclude a darkness-to-shape transform kernel 36 disposed in-line betweenthe wavelet transform unit 35 and the artificial neural network 38 totransform the sub-band images into a shape-based sub-band image, whereinthe artificial neural network 38 may be trained on the shape-basedsub-band images by deep learning.

Specifically, FIG. 2(a) shows an embodiment in which a darkness-to-shapetransform kernel 36 transforming the sub-band images into oneshape-based sub-band image is disposed in-line between the wavelettransform unit 35 and the artificial neural network 38, such that oneartificial neural network 38 is trained on the one shape-based sub-bandimage by deep learning, and FIG. 2(b) shows an embodiment in which adarkness-to-shape transform kernel 36 transforming the sub-band imagesinto respective shape-based sub-band images is disposed in-line betweenthe wavelet transform unit 35 and the artificial neural network 36, suchthat plural independent artificial neural networks 38 are trained on therespective shape-based sub-band images by deep learning.

In addition, referring to FIG. 2(b), a support vector machine 39performing machine learning may be further disposed downstream of theartificial neural networks 38 to collect results of analysis by theindependent artificial neural networks 38 to determine the grade ofliver steatosis, wherein the support vector machine 39 conductsclassification of liver steatosis based on feature vectors received fromthe artificial neural networks 38.

The darkness-to-shape transform kernel 36 shown in FIG. 2(b) is adaptedto transform each of the sub-band images generated by the wavelettransform unit 35 into a shape-based sub-band image, and may include afirst kernel 36 a transforming a sub-band LL image into a shape-basedsub-band LL image; a second kernel 36 b transforming a sub-band LH imageinto a shape-based sub-band LH image; a third kernel 36 c transforming asub-band HL image into a shape-based sub-band HL image; and a fourthkernel 36 d transforming the sub-band HH image into a shape-basedsub-band HH image.

The artificial neural network 38 shown in FIG. 2(b) may include: a firstartificial neural network 38 a trained on the shape-based sub-band LLimage; a second artificial neural network 38 b trained on theshape-based sub-band LH image; a third artificial neural network 38 ctrained on the shape-based sub-band HL image; and a fourth artificialneural network 38 d trained on the shape-based sub-band HH image.

FIG. 3 is a diagram illustrating a process of acquiring a pattern imageand an integrated image by the image integration unit 34. First, theregion-of-interest extraction unit 32 acquires a first image 22 a byextracting a region including a kidney and a liver from an ultrasoundimage 22 in a parasagittal scan plane using an elliptical window,acquires a second image 22 b by extracting only a kidney region from thefirst image 22 a, acquires a kidney border image 22 c by extractionalong a boundary between the first image 22 a and the second image 22 b,and acquires a third image 22 d by extracting the remaining portion ofthe first image 22 a excluding the second image 22 b and the kidneyborder image 22 c.

Then, the image integration unit 34 generates a pattern image 22 e ofthe second image 22 b by sequentially reading image pixels in the secondimage 22 b while circularly moving starting from center coordinates ofthe second image 22 b to form a rectangular image 22 e. In addition, theimage integration unit 34 generates a pattern image 22 f of the kidneyborder image 22 c by sequentially reading image pixels in the kidneyborder image 22 c while circularly moving along the kidney border image22 c to form a rectangular image 22 f.

Further, the image integration unit 34 generates a pattern image 22 g ofthe third image 22 d by sequentially reading image pixels in the thirdimage 22 d while scanning the third image from left to right and top tobottom to form a rectangular image. Then, the image integration unit 34combines the generated pattern images 22 e, 22 f, 22 g into anintegrated image 22 h.

An image pattern in the integrated image 22 h significantly differsbetween different liver steatosis grades and thus can be efficientlylearned and recognized by the artificial neural network.

FIG. 4 is a block diagram of the region-of-interest extraction unit 32extracting a region of interest from an ultrasound image acquired by theultrasound probe sensor 30 according to one embodiment of the presentinvention. The region-of-interest extraction unit 32 according to thisembodiment includes: an organ image database 42 storing reference organimages as a reference for organ matching; a reference feature pointstorage unit 65 storing a feature point of each of the reference organimages; a feature point detection unit 62 extracting matching featurepoints from the ultrasound image; a feature point pair generation unit64 comparing the reference feature point with the matching feature pointof each patch image, which is formed around the matching feature point,to select patch images having a matching feature point with highsimilarity to the reference feature point and generating a pair offeature points between the reference organ image and each of theselected patch images; an outlier detection unit 49 eliminating abnormalpairs of feature points among the generated pairs of feature points; ageometric parameter calculation unit 66 calculating geometric parametersrequired for registration between the resultant pair of feature points,the geometric parameters including translation, rotation and scaling;and an organ matching unit 68 acquiring a registered patch image usingthe geometric parameters, calculating a cross-correlation coefficientbetween the registered patch image and the reference organ image,determining that a corresponding organ is located in a current patchimage in the ultrasound image if the cross-correlation coefficient isgreater than a predetermined threshold, extracting an organ image regionat a corresponding location, and transmitting the organ image region tothe image integration unit 34.

The organ image region may include at least one selected from the groupof the first image, the second image obtained by extracting only akidney region from the first image, the kidney border image obtained byextraction along the boundary between the first image and the secondimage, and the third image obtained by extracting the remaining regionof the first image excluding the second image and the kidney borderimage.

FIG. 5 shows an exemplary ultrasound image in which different organs arelabeled with different colors, the ultrasound image being obtained bysemantic segmentation of an ultrasound image 22 in a parasagittal scanplane.

In FIG. 5, reference numeral 26 denotes a liver, reference numeral 25denotes a spleen, reference numeral 24 denotes a kidney, and referencenumeral 23 denotes a diaphragm.

FIG. 6 is a diagram of the feature point detection unit 62 selectingoverlapping pixels between sub-band frame images as a feature pointaccording to one embodiment of the present invention. The feature pointdetection unit 62 includes: an ultrasound probe sensor 30 acquiring anultrasound image from an examination area of a patient; a wavelet frameunit 100 acquiring a sub-band HL frame image, a sub-band LH frame image,and a sub-band HH frame image by wavelet frame transform of theultrasound image; feature point determination units 102, 104, 106applying a feature point determination threshold to the respectivesub-band frame images on a pixel-by-pixel basis; and an overlappingpixel determination unit 120 detecting overlapping pixels bymultiplication between the sub-band frame images with respect to pixelshaving a value greater than the feature point determination threshold.In FIG. 6, corner points (indicated by ‘+’ in the drawing) can be foundin a resultant image 122 from the overlapping pixel determination unit120.

Reference numeral 130 denotes an exemplary ultrasound image having thecorner points superimposed thereon. The feature point pair generationunit 64 generates pairs of feature points composed of the corner points(indicated by ‘+’) and corresponding feature points stored in thereference feature point storage unit 65, and the geometric parametercalculation unit 66 calculates geometric parameters required forregistration between the generated pair of feature points, wherein thegeometric parameters may include translation, rotation, and scaling.

FIG. 7 shows a sub-band LL frame image 85, a sub-band HL frame image 86,a sub-band LH frame image 87, and a sub-band HH frame image 88 acquiredby applying wavelet frame transform to an ultrasound image 84.

Since wavelet frame transform does not include a subsampling process,image size does not change before and after transform.

FIG. 8 is a diagram of an apparatus for AI-based automatic ultrasounddiagnosis of liver steatosis according to a further embodiment of thepresent invention, wherein an ultrasound medical device 60 is connectedto a remote medical diagnosis system 300 such that an AI-based virtualdoctor 99 residing in the remote medical diagnosis system automaticallyanalyzes medical image data of a patient to perform diagnosis. Theremote medical diagnosis system 300 may include: an ultrasound medicaldevice 60 provided with an ultrasound probe sensor 30; a wirelesstransmitter 60 a integrated in the ultrasound medical device 60 towirelessly transmit medical image data of a patient measured by theultrasound medical device 60; the apparatus for AI-based automaticultrasound diagnosis of liver steatosis 600 according to the presentinvention, the apparatus receiving an ultrasound image of the patientfrom the wireless transmitter 60 a and automatically determining a gradeof liver steatosis; a user terminal 400 including a camera 61 monitoringuse of the ultrasound medical device 60, a first authentication unit 93wirelessly authenticating product ID of the ultrasound medical device60, a recording unit 94 storing the ultrasound image of the patientobtained by the ultrasound medical device 60, an Internet connector 96transmitting the ultrasound image and the product ID of the ultrasoundmedical device 60 to a remote diagnosis server 81 via a communicationnetwork 80 and providing a communication channel for a remoteconsultation service, and a first consultation service unit providing aconsultation service with a medical expert; a communication interface105 providing a connection to the apparatus for AI-based automaticultrasound diagnosis of liver steatosis 600 and the user terminal 400;an artificial neural network 90 residing as software in the userterminal 400 and trained on a medical image database accumulated by theultrasound medical device 60 by deep learning; a virtual doctor 99residing as software in the user terminal 400 and including a guide unit91 guiding or instructing how to use the ultrasound medical device 60and a diagnosis unit 92 outputting a diagnostic result obtained byautomatic analysis of the medical image data of the patient obtained bythe ultrasound medical device 60 using the deep learning-trainedartificial neural network 90; and a medical expert terminal 200including a receiver (not shown) receiving the medical image data andthe ultrasound image via the communication network 80 and a secondconsultation service unit providing a consultation service between auser and a medical expert.

The guide unit 91 serves to guide or instruct a user on how to use theultrasound medical device 60 based on results of monitoring use of theultrasound medical device 60 in real time using the camera 61.

The medical expert terminal 200 may further include a camera 14, amicrophone 15, and a mouse 11.

Next, based on the details described above, an apparatus for AI-basedautomatic ultrasound diagnosis of liver steatosis according to variousembodiments of the present invention will be briefly discussed.

An apparatus for AI-based automatic ultrasound diagnosis of liversteatosis 600 according to one embodiment of the present invention mayinclude a probe sensor 30, a region-of-interest extraction unit 32, animage integration unit 34, a darkness-to-shape transform kernel 36, andan artificial neural network 38.

The ultrasound probe sensor 30 may acquire an ultrasound image from apatient.

The region-of-interest extraction unit 32 may acquire images of one ormore regions helpful to diagnosis of liver steatosis from the ultrasoundimage. The images of the regions of interest may be used in diagnosis ofliver steatosis. The region-of-interest extraction unit 32 may extractthe images of the regions of interest from the ultrasound image acquiredby the ultrasound probe sensor 30. The images extracted or generated bythe region-of-interest extraction unit 32 to be transmitted to the imageintegration unit 34 are designated by reference numeral 47 in FIG. 1 toFIG. 4.

The image integration unit 34 may combine the images of the one or moreregions of interest into one integrated image. For example, if there isonly an image of one region of interest, the image itself may be used asthe integrated image. If there are images of plural regions of interest,the images may be combined into one integrated image.

The darkness-to-shape transform kernel 36 may transform the integratedimage into a shape-based integrated image.

The artificial neural network 38 may be trained in advance on theshape-based integrated image by deep learning. In addition, theartificial neural network 38 trained in advance on the shape-basedintegrated image by deep learning may automatically determine a grade ofliver steatosis based on the ultrasound image of the patient receivedfrom the ultrasound probe sensor 30.

In addition, an apparatus for AI-based automatic ultrasound diagnosis ofliver steatosis 600 according to another embodiment of the presentinvention may include an ultrasound probe sensor 30, aregion-of-interest extraction unit 32, an image integration unit 34, andan artificial neural network 38, wherein a darkness-to-shape transformkernel 36 may be omitted.

The image integration unit 34 may include a rearrangement unit 34 a. Therearrangement unit 34 a rearranges the images of the one or more regionsof interest acquired by the region-of-interest extraction unit 32 toacquire pattern images. The image integration unit 34 may combine thepattern images into one integrated image.

If there are images of plural regions of interest, the rearrangementunit 34 a may rearrange the images into respective pattern images.

For example, the rearrangement unit 34 a may receive a first image 22 a,a second image 22 b, a kidney border image 22 c, and a third image 22 dfrom the region-of-interest extraction unit 32, the first image beingobtained by extracting a region including a liver and a kidney from theultrasound image, the second image being obtained by extracting only akidney region from the first image, the kidney border image beingobtained by extraction along a boundary between the first image and thesecond image, and the third image being 22 d being obtained byextracting the remaining portion of the first image excluding the secondimage and the kidney border image. Then, the rearrangement unit 34 a maygenerate a pattern image 22 e of the second image 22 b by taking imagepixels from the second image 22 b while circularly moving starting fromcenter coordinates of the second image 22 b and rearranging the takenimage pixels into a rectangular image, generate a pattern image 22 f ofthe kidney border image 22 c by taking image pixels from the kidneyborder image 22 c while circularly moving along the kidney border image22 c and rearranging the taken image pixels into a rectangular image,and generate a pattern image 22 g of the third image 22 d by takingimage pixels from the third image 22 d by pixel scanning and rearrangingthe taken image pixels into a rectangular image.

Here, a region extracted from the ultrasound image by theregion-of-interest extraction unit 32 may be selected from among regionsof the organs of interest described above, including a liver region anda kidney region.

As used herein, the expression “taking image pixels from the secondimage, the kidney border image, and the third image” may mean that therearrangement unit 34 a reads or scans the image pixels. In addition,the expression “taking image pixels from the third image by pixelscanning” may mean scanning the image pixels according to apredetermined pixel scanning scheme, as described above.

It should be understood that the image pixels taken by the rearrangementunit 34 a may be rearranged into images having various shapes such as acircular shape, without being limited to a rectangular image.

In addition, an apparatus for AI-based automatic ultrasound diagnosis ofliver steatosis 600 according to a further embodiment of the presentinvention may include an ultrasound probe sensor 30, aregion-of-interest extraction unit 32, an image integration unit 34, awavelet transform unit 35, and an artificial neural network 38.

The wavelet transform unit 35 may decompose the integrated imagegenerated by the image integration unit 34 into sub-band images bydiscrete wavelet transform. Here, the sub-band images include a sub-bandLL image, a sub-band LH image, a sub-band HL image, and a sub-band HHimage.

The artificial neural network 38 may be trained in advance on thesub-band images generated by the wavelet transform unit 35 by deeplearning. In addition, the artificial neural network 38 trained inadvance on the sub-band images by deep learning may automaticallydetermine a grade of liver steatosis based on an ultrasound image of apatient received from the ultrasound probe sensor 30. Wavelet transformcan improve effectiveness in learning and accuracy of determination.

In addition, the apparatus for AI-based automatic ultrasound diagnosisof liver steatosis 600 according to this embodiment may further includea darkness-to-shape transform kernel 36.

Here, the darkness-to-shape transform kernel 36 may transform thesub-band images into a shape-based sub-band image. In addition, theartificial neural network 38 may be trained in advance on theshape-based sub-band image by deep learning. The deep learning-trainedartificial neural network 38 may automatically determine a grade ofliver steatosis in a patient based on the shape-based sub-band image.

The region-of-interest extraction unit 32 according to the presentinvention may extract images of at least two selected from the group ofliver, right portal vein (RPV), hepatic vein, kidney, spleen, anddiaphragm regions from an ultrasound image. Here, the liver, the hepaticvein, the kidney, the spleen, and the diaphragm may be included in theorgan of interest described above and may be replaced with other organsthat can be used in diagnosis of liver steatosis.

In addition, the region-of-interest extraction unit 32 according to oneembodiment of the present invention may include an organ image database42, a reference feature point storage unit 65, a feature point detectionunit 62, a feature point pair generation unit 64, an outlier detectionunit 49, a geometric parameter calculation unit 66, and an organmatching unit 68.

The organ image database 42 may store reference organ images as areference for organ matching.

The reference feature point storage unit 65 may store feature points ofthe reference organ images.

The feature point detection unit 62 may extract matching feature pointsfrom the ultrasound image.

The feature point pair generation unit 64 may compare the referencefeature point with the matching feature point of each patch image, whichis formed around the matching feature point, to select patch imageshaving a matching feature point with high similarity to the referencefeature point to generate a pair of feature points between the referenceorgan image and each of the selected patch images. Here, the similarityto the reference feature point may be determined, for example, bycreating a vector between the reference feature point and the matchingfeature point, calculating the length of the vector, and comparing thecalculated vector length with a predetermined value.

The outlier detection unit 49 may eliminate abnormal pairs of featurepoints among the generated pairs of feature points. For example, whenthe length of a vector between a pair of feature points exceeds apredetermined outlier threshold, the pair of feature points may beeliminated.

The geometric parameter calculation unit 66 may calculate geometricparameters required for registration between the pair of feature pointsobtained by the feature point pair generation unit 64, wherein thegeometric parameters may include translation, rotation, and scaling.

The organ matching unit 68 may acquire a registered patch image usingthe geometric parameters calculated by the geometric parametercalculation unit 66, calculate a cross-correlation coefficient betweenthe registered patch image and the reference organ image, determine thata corresponding organ is located in a current patch image in theultrasound image if the cross-correlation coefficient is greater than apredetermined threshold, extract an organ image region at acorresponding location, and transmit the organ image region to the imageintegration unit. In FIG. 4, reference numeral 47 may denote the organimage region transmitted to the image integration unit 34 from the organmatching unit 68.

In another embodiment, the region-of-interest extraction unit 32 mayinclude an artificial neural network performing semantic segmentation onthe ultrasound image to acquire a semantic segmented ultrasound image inwhich different organs of interest are labeled with different values ordifferent colors for extraction of the images of the regions ofinterest. The region-of-interest extraction unit 32 may generate a firstimage by extracting a region including a kidney and a liver from theultrasound image by semantic segmentation, generate a second image byextracting only a kidney region from the first image by semanticsegmentation, generate a kidney border image by extraction along aboundary between the first image and the second image, generate a thirdimage by extracting the remaining portion of the first image excludingthe second image and the kidney border image, and transmit the secondimage, the kidney border image, and the third image to the imageintegration unit.

Here, the kidney and the liver may be replaced with other organs asdescribed above.

In a further embodiment, the region-of-interest extraction unit 32 mayinclude a wavelet frame unit 100, a feature point detection unit 62, afeature point pair generation unit 64, a geometric parameter calculationunit 66, and an organ matching unit 68. Here, the feature pointdetection unit 62 may include feature point determination units 102,104, 106 and an overlapping pixel determination unit 120.

The wavelet frame unit 100 may perform wavelet frame transform on thereference organ image and the ultrasound image to acquire a sub-band HLframe image, a sub-band LH frame image, and a sub-band HH frame image.

The feature point determination units 102, 104, 106 may apply a featurepoint determination threshold to the respective sub-band frame images ona pixel-by-pixel basis.

The overlapping pixel determination unit 120 may find overlapping pixelsby multiplication between the sub-band frame images with respect topixels having a value greater than the feature point determinationthreshold.

The feature point detection unit 62 may select overlapping pixels of thereference organ image obtained by the overlapping pixel determinationunit 120 as the reference feature point and select overlapping pixels ofthe ultrasound image obtained by the overlapping pixel determinationunit 120 as the matching feature point.

An apparatus for AI-based automatic ultrasound diagnosis of liversteatosis according to a further embodiment of the present invention(hereinafter referred to as “apparatus for AI-based automatic ultrasounddiagnosis of liver steatosis including a remote medical diagnosissystem”) may include a remote medical diagnosis system including anultrasound medical device 60, the apparatus for AI-based automaticultrasound diagnosis of liver steatosis 600 according to the aboveembodiments, a communication interface (not shown), a user terminal 400,and a medical expert terminal 200.

The ultrasound medical device 60 may include an ultrasound probe sensor30 and a wireless transmitter 60 a. The wireless transmitter 60 a may beintegrated in the ultrasound medical device 60 to wirelessly transmitmedical image data of a patient measured by the ultrasound medicaldevice 60.

The apparatus for AI-based automatic ultrasound diagnosis of liversteatosis 600 may receive an ultrasound image of the patient from thewireless transmitter 60 a and automatically determine a grade of liversteatosis. The apparatus for AI-based automatic ultrasound diagnosis ofliver steatosis including the remote medical diagnosis system mayinclude the apparatus for AI-based automatic ultrasound diagnosis ofliver steatosis 600 according to the above embodiments of the presentinvention.

The communication interface (not shown) may provide a connection to theapparatus for AI-based automatic ultrasound diagnosis of liver steatosis600 and the user terminal 400. For example, data such as the integratedimage, the pattern image, and the liver steatosis grade of the patientmay be transmitted to the user terminal 400 from the apparatus forAI-based automatic ultrasound diagnosis of liver steatosis 600 via thecommunication interface.

The user terminal 400 may include a camera 61, a first authenticationunit 93, a recording unit 94, an internet connector 96, and a firstconsultation service unit 95. Here, the camera 61 may monitor use of theultrasound medical device 60. The first authentication unit 93 maywirelessly authenticate a product ID of the ultrasound medical device60. The recording unit 94 may store the ultrasound image of the patientacquired by the ultrasound medical device 60. In addition, the recordingunit 94 may store the data transmitted to the user terminal 400 from theapparatus for AI-based automatic ultrasound diagnosis of liver steatosis600. The internet connector 64 may transmit the ultrasound image and theproduct ID of the ultrasound medical device 60 to a remote diagnosisserver 81 via a communication network 80 and may provide a communicationchannel for a remote consultation service. The first consultationservice unit 95 may provide a consultation service with a medicalexpert.

In addition, the user terminal 400 may further include an artificialneural network 90 and a virtual doctor 99 including a guide unit 91 anda diagnosis unit 92. Here, the artificial neural network 90 and thevirtual doctor 99 may reside as software in the user terminal 400.

The artificial neural network 90 may be trained on a medical imagedatabase accumulated by the ultrasound medical device 60 by deeplearning.

The guide unit 91 may guide or instruct how to use the ultrasoundmedical device 60 and the diagnosis unit 92 may output a diagnosticresult obtained by automatic analysis of the medical image data of thepatient acquired by the ultrasound medical device 60 using the deeplearning-trained artificial neural network 90.

The medical expert terminal 200 may include a receiver (not shown) and asecond consultation service unit (not shown).

The receiver (not shown) may receive the medical image data or theultrasound image via the communication network 80 and the secondconsultation service unit (not shown) may provide a consultation servicebetween a user and a medical expert.

A remote medical diagnosis method according to one embodiment of thepresent invention may include the steps of: extracting images of one ormore regions of interest helpful to diagnosis of liver steatosis from anultrasound image of a patient; generating one integrated image byconcatenating and combining the images of the one or more regions ofinterest into one; transforming the integrated image into a shape-basedintegrated image or a pattern image; training an artificial neuralnetwork on the shape-based integrated image or the pattern image by deeplearning; automatically determining, by a virtual doctor, a grade ofliver steatosis based on the ultrasound image of the patient; andproviding, by a remote medical diagnosis system, a remote consultationservice with a medical expert.

The remote medical diagnosis method according to the present inventionmay be carried out based on the various embodiments of the apparatus forAI-based automatic ultrasound diagnosis of liver steatosis and theremote medical diagnosis system set forth above.

The remote medical diagnosis method according to the present inventionmay be realized in the form of program instructions which can beimplemented through various computer components, and may be recorded ina computer-readable storage medium. The computer-readable storage mediummay include program instructions, a data file, a data structure, and thelike either alone or in combination thereof. The program instructionsrecorded in the computer-readable storage medium may be any programinstructions particularly designed and structured for the presentinvention or known to those skilled in the field of computer software.Examples of the computer-readable storage medium include magneticrecording media, such as hard disks, floppy disks and magnetic tapes,optical data storage media, such as CD-ROMs and DVD-ROMs,magneto-optical media such as floptical disks, and hardware devices,such as read-only memories (ROMs), random-access memories (RAMs), andflash memories, which are particularly structured to store and implementthe program instructions. Examples of the program instructions includenot only assembly language code formatted by a compiler but alsohigh-level language code which can be implemented by a computer using aninterpreter. The hardware device described above may be configured tooperate as one or more software modules to perform operations of thepresent invention, and vice versa.

In addition, the remote medical diagnosis method may be implemented inthe form of a computer-executable computer program or application storedin a recording medium.

Although some embodiments have been described herein, it should beunderstood that these embodiments are provided for illustration and thatvarious modifications, changes, alterations, and equivalent embodimentscan be made by those skilled in the art without departing from thespirit and scope of the invention. Therefore, the embodiments are not tobe construed in any way as limiting the present invention. For example,each component described as a single type may be implemented in adistributed manner, and, similarly, components described as distributedmay be implemented in a combined form.

The scope of the present application should be defined by the appendedclaims and equivalents thereof rather than by the detailed description,and all changes or modifications derived from the spirit and scope ofthe claims and equivalents thereof should be construed as within thescope of the present invention.

LIST OF REFERENCE NUMERALS

-   -   600: Apparatus for AI-based automatic ultrasound diagnosis of        liver steatosis    -   30: Ultrasound probe sensor    -   32: Region-of-interest extraction unit    -   34: Image integration unit    -   34 a: Rearrangement unit    -   36: Darkness-to-shape transform kernel    -   38: Artificial neural network    -   35: Wavelet transform unit

What is claimed is:
 1. An apparatus for AI-based automatic ultrasounddiagnosis of liver steatosis, comprising: an ultrasound probe sensoracquiring an ultrasound image from a patient; a region-of-interestextraction unit acquiring images of one or more regions of interesthelpful to diagnosis of liver steatosis from the ultrasound image; animage integration unit combining the images of the one or more regionsof interest into one integrated image; a darkness-to-shape transformkernel transforming the integrated image into a shape-based integratedimage; and an artificial neural network trained on the shape-basedintegrated image by deep learning, wherein the deep learning-trainedartificial neural network automatically determines a grade of liversteatosis in the patient based on the ultrasound image of the patientreceived from the ultrasound probe sensor.
 2. An apparatus for AI-basedautomatic ultrasound diagnosis of liver steatosis, comprising: anultrasound probe sensor acquiring an ultrasound image from a patient; aregion-of-interest extraction unit acquiring images of one or moreregions of interest helpful to diagnosis of liver steatosis from theultrasound image; an image integration unit comprising a rearrangementunit rearranging the images of the one or more regions of interestacquired by the region-of-interest extraction unit to acquire a patternimage, the image integration combining the rearranged images into oneintegrated image; and an artificial neural network trained on theintegrated image by deep learning, wherein the deep learning-trainedartificial neural network automatically determines a grade of liversteatosis in the patient based on the ultrasound image of the patientreceived from the ultrasound probe sensor.
 3. The apparatus accordingclaim 2, wherein the rearrangement unit receives a first image, a secondimage, a kidney border image, and a third image from theregion-of-interest extraction unit, the first image being obtained byextracting a kidney region and a liver region from the ultrasound image,the second image being obtained by extracting only the kidney regionfrom the first image, the kidney border image being obtained byextraction along a boundary between the first image and the secondimage, and the third image being obtained by extracting the remainingportion of the first image excluding the second image and the kidneyborder image; generates a pattern image of the second image by takingimage pixels from the second image while circularly moving starting fromcenter coordinates of the second image and rearranging the taken imagepixels into a rectangular image; generates a pattern image of the kidneyborder image taking image pixels from the kidney border image whilecircularly moving along the kidney border image and rearranging thetaken image pixels into a rectangular image; and generates a patternimage of the third image by taking image pixels from the third image bypixel scanning and rearranging the taken images into a rectangularimage.
 4. An apparatus for AI-based automatic ultrasound diagnosis ofliver steatosis, comprising: an ultrasound probe sensor acquiring anultrasound image from a patient; a region-of-interest extraction unitextracting images of one or more regions of interest from the ultrasoundimage; an image integration unit combining the images of the images ofthe one or more regions of interest into one integrated image; a wavelettransform unit decomposing the integrated image into sub-band images bydiscrete wavelet transform; and an artificial neural network trained onthe sub-band images by deep learning, wherein the deep learning-trainedartificial neural network automatically determines a grade of liversteatosis in the patient based on the ultrasound image of the patientreceived from the ultrasound probe sensor.
 5. The apparatus according toclaim 4, further comprising: a darkness-to-shape transform kerneltransforming the sub-band images into a shape-based sub-band image,wherein the artificial neural network is trained on the shape-basedsub-band image by deep learning.
 6. The apparatus according to claim 1,wherein the region-of-interest extraction unit extracts at least twoselected from the group of images of liver, right portal vein (RPV),hepatic vein, kidney, spleen, and diaphragm regions.
 7. The apparatusaccording to claim 1, wherein the region-of-interest extraction unitcomprises: an organ image database storing reference organ images as areference for organ matching; a reference feature point storage unitstoring feature points of the reference organ image (hereinafterreferred to as “reference feature point”); a feature point detectionunit extracting matching feature points from the ultrasound image; afeature point pair generation unit comparing the reference feature pointwith respective matching feature points in patch images, which areformed around the respective matching feature points, to select patchimages having a matching feature point with high similarity to thereference feature point and generating a pair of feature points betweenthe reference organ image and each of the selected patch images; anoutlier detection unit eliminating abnormal pairs of feature pointsamong the generated pairs of feature points; a geometric parametercalculation unit calculating geometric parameters required forregistration of the resultant pairs of feature points, the geometricparameters comprising translation, rotation, and scaling; and an organmatching unit acquiring a registered patch image using the geometricparameters, calculating a cross-correlation coefficient between theregistered patch image and the reference organ image, determining that acorresponding organ is located in a current patch image in theultrasound image if the cross-correlation coefficient is greater than apredetermined threshold, extracting an organ image region at acorresponding location, and transmitting the organ image region to theimage integration unit.
 8. The apparatus according to claim 1, whereinthe region-of-interest extraction unit comprises an artificial neuralnetwork performing semantic segmentation on the ultrasound image toacquire a semantic segmented ultrasound image in which different organsof interest are labeled with different values or different colors forextraction of the images of the regions of interest, theregion-of-interest extraction unit generating a first image byextracting a region including a kidney and a liver from the ultrasoundimage by semantic segmentation, generating a second image by extractingonly a kidney region from the first image by semantic segmentation,generating a kidney border image by extraction along a boundary betweenthe first image and the second image, generating a third image byextracting the remaining portion of the first image excluding the secondimage and the kidney border image, and transmitting the second image,the kidney border image, and the third image to the image integrationunit.
 9. The apparatus according to claim 1, wherein theregion-of-interest extraction unit comprises: a wavelet frame unitacquiring a sub-band HL frame image, a sub-band LH frame image, and asub-band HH frame image by wavelet frame transform of the referenceorgan image and the ultrasound image; a feature point detection unitcomprising feature point determination units applying a feature pointdetermination threshold to the respective sub-band frame images on apixel-by-pixel basis and an overlapping pixel determination unitdetecting overlapping pixels by performing pixel-by-pixel multiplicationbetween the sub-band frame images with respect to pixels having a valuegreater than the feature point determination threshold, the featurepoint detection unit selecting overlapping pixels of the reference organimage acquired by the overlapping pixel determination unit as areference feature point and selecting overlapping pixels of theultrasound image acquired by the overlapping pixel determination unit asa matching feature point; a feature point pair generation unit comparingthe reference feature point with respective matching feature points inpatch images, which are formed around the respective matching featurepoints, to select patch images having a matching feature point with highsimilarity to the reference feature point and generating a pair offeature points between the reference organ image and each of theselected patch images; an outlier detection unit eliminating abnormalpairs of feature points among the generated pairs of feature points; ageometric parameter calculation unit calculating geometric parametersrequired for registration of the resultant pair of feature points, thegeometric parameters comprising translation, rotation, and scaling; andan organ matching unit acquiring a registered patch image using thegeometric parameters, calculating a cross-correlation coefficientbetween the reference organ image and the registered patch image,determining that a corresponding organ is located in a current patchimage in the ultrasound image if the cross-correlation coefficient isgreater than a predetermined threshold, extracting an organ image regionat a corresponding location, and transmitting the organ image region tothe image integration unit.
 10. An apparatus for AI-based automaticultrasound diagnosis of liver steatosis comprising a remote medicaldiagnosis system, the remote medical diagnosis system comprising: anultrasound medical device provided with an ultrasound probe sensor; awireless transmitter integrated in the ultrasound medical device towirelessly transmit medical image data of a patient measured by theultrasound medical device; an apparatus for AI-based automaticultrasound diagnosis of liver steatosis receiving an ultrasound image ofthe patient from the wireless transmitter and automatically determininga grade of liver steatosis, the apparatus being the apparatus forAI-based automatic ultrasound diagnosis of liver steatosis accordingclaim 1; a user terminal comprising a camera monitoring use of theultrasound medical device, a first authentication unit wirelesslyauthenticating a product ID of the ultrasound medical device, arecording unit storing the ultrasound image of the patient acquired bythe ultrasound medical device, an Internet connector transmitting theultrasound image and the product ID of the ultrasound medical device toa remote diagnosis server via a communication network and providing acommunication channel for a remote consultation service, and a firstconsultation service unit providing a consultation service with amedical expert; a communication interface providing a connection to theapparatus and the user terminal; an artificial neural network residingas software in the user terminal and trained on a medical image databaseaccumulated by the ultrasound medical device by deep learning; a virtualdoctor residing as software in the user terminal and comprising a guideunit guiding or instructing how to use the ultrasound medical device anda diagnosis unit outputting a diagnostic result obtained by automaticanalysis of the medical image data of the patient obtained by theultrasound medical device using the deep learning trained artificialneural network; and a medical expert terminal comprising a receiverreceiving the medical image data and the ultrasound image via thecommunication network and a second consultation service unit providing aconsultation service between a user and a medical expert.
 11. A remotemedical diagnosis method using the apparatus for AI-based automaticultrasound diagnosis of liver steatosis according to claim 10, theremote medical diagnosis method comprising the steps of: extractingimages of one or more regions of interest helpful to diagnosis of liversteatosis from the ultrasound image; concatenating and combining theimages of the one or more regions of interest into one integrated image;transforming the integrated image into a shape-based integrated image ora pattern image; training the artificial neural network on theshape-based integrated image or the pattern image by deep learning;automatically determining, by the virtual doctor, a grade of liversteatosis based on the ultrasound image of the patient; and providing,by the remote medical diagnosis system, a remote consultation servicewith a medical expert.
 12. The apparatus according to claim 2, whereinthe region-of-interest extraction unit extracts at least two selectedfrom the group of images of liver, right portal vein (RPV), hepaticvein, kidney, spleen, and diaphragm regions.
 13. The apparatus accordingto claim 2, wherein the region-of-interest extraction unit comprises: anorgan image database storing reference organ images as a reference fororgan matching; a reference feature point storage unit storing featurepoints of the reference organ image (hereinafter referred to as“reference feature point”); a feature point detection unit extractingmatching feature points from the ultrasound image; a feature point pairgeneration unit comparing the reference feature point with respectivematching feature points in patch images, which are formed around therespective matching feature points, to select patch images having amatching feature point with high similarity to the reference featurepoint and generating a pair of feature points between the referenceorgan image and each of the selected patch images; an outlier detectionunit eliminating abnormal pairs of feature points among the generatedpairs of feature points; a geometric parameter calculation unitcalculating geometric parameters required for registration of theresultant pairs of feature points, the geometric parameters comprisingtranslation, rotation, and scaling; and an organ matching unit acquiringa registered patch image using the geometric parameters, calculating across-correlation coefficient between the registered patch image and thereference organ image, determining that a corresponding organ is locatedin a current patch image in the ultrasound image if thecross-correlation coefficient is greater than a predetermined threshold,extracting an organ image region at a corresponding location, andtransmitting the organ image region to the image integration unit. 14.The apparatus according to claim 2, wherein the region-of-interestextraction unit comprises an artificial neural network performingsemantic segmentation on the ultrasound image to acquire a semanticsegmented ultrasound image in which different organs of interest arelabeled with different values or different colors for extraction of theimages of the regions of interest, the region-of-interest extractionunit generating a first image by extracting a region including a kidneyand a liver from the ultrasound image by semantic segmentation,generating a second image by extracting only a kidney region from thefirst image by semantic segmentation, generating a kidney border imageby extraction along a boundary between the first image and the secondimage, generating a third image by extracting the remaining portion ofthe first image excluding the second image and the kidney border image,and transmitting the second image, the kidney border image, and thethird image to the image integration unit.
 15. The apparatus accordingto claim 2, wherein the region-of-interest extraction unit comprises: awavelet frame unit acquiring a sub-band HL frame image, a sub-band LHframe image, and a sub-band HH frame image by wavelet frame transform ofthe reference organ image and the ultrasound image; a feature pointdetection unit comprising feature point determination units applying afeature point determination threshold to the respective sub-band frameimages on a pixel-by-pixel basis and an overlapping pixel determinationunit detecting overlapping pixels by performing pixel-by-pixelmultiplication between the sub-band frame images with respect to pixelshaving a value greater than the feature point determination threshold,the feature point detection unit selecting overlapping pixels of thereference organ image acquired by the overlapping pixel determinationunit as a reference feature point and selecting overlapping pixels ofthe ultrasound image acquired by the overlapping pixel determinationunit as a matching feature point; a feature point pair generation unitcomparing the reference feature point with respective matching featurepoints in patch images, which are formed around the respective matchingfeature points, to select patch images having a matching feature pointwith high similarity to the reference feature point and generating apair of feature points between the reference organ image and each of theselected patch images; an outlier detection unit eliminating abnormalpairs of feature points among the generated pairs of feature points; ageometric parameter calculation unit calculating geometric parametersrequired for registration of the resultant pair of feature points, thegeometric parameters comprising translation, rotation, and scaling; andan organ matching unit acquiring a registered patch image using thegeometric parameters, calculating a cross-correlation coefficientbetween the reference organ image and the registered patch image,determining that a corresponding organ is located in a current patchimage in the ultrasound image if the cross-correlation coefficient isgreater than a predetermined threshold, extracting an organ image regionat a corresponding location, and transmitting the organ image region tothe image integration unit.
 16. The apparatus according to claim 4,wherein the region-of-interest extraction unit extracts at least twoselected from the group of images of liver, right portal vein (RPV),hepatic vein, kidney, spleen, and diaphragm regions.
 17. The apparatusaccording to claim 4, wherein the region-of-interest extraction unitcomprises: an organ image database storing reference organ images as areference for organ matching; a reference feature point storage unitstoring feature points of the reference organ image (hereinafterreferred to as “reference feature point”); a feature point detectionunit extracting matching feature points from the ultrasound image; afeature point pair generation unit comparing the reference feature pointwith respective matching feature points in patch images, which areformed around the respective matching feature points, to select patchimages having a matching feature point with high similarity to thereference feature point and generating a pair of feature points betweenthe reference organ image and each of the selected patch images; anoutlier detection unit eliminating abnormal pairs of feature pointsamong the generated pairs of feature points; a geometric parametercalculation unit calculating geometric parameters required forregistration of the resultant pairs of feature points, the geometricparameters comprising translation, rotation, and scaling; and an organmatching unit acquiring a registered patch image using the geometricparameters, calculating a cross-correlation coefficient between theregistered patch image and the reference organ image, determining that acorresponding organ is located in a current patch image in theultrasound image if the cross-correlation coefficient is greater than apredetermined threshold, extracting an organ image region at acorresponding location, and transmitting the organ image region to theimage integration unit.
 18. The apparatus according to claim 4, whereinthe region-of-interest extraction unit comprises an artificial neuralnetwork performing semantic segmentation on the ultrasound image toacquire a semantic segmented ultrasound image in which different organsof interest are labeled with different values or different colors forextraction of the images of the regions of interest, theregion-of-interest extraction unit generating a first image byextracting a region including a kidney and a liver from the ultrasoundimage by semantic segmentation, generating a second image by extractingonly a kidney region from the first image by semantic segmentation,generating a kidney border image by extraction along a boundary betweenthe first image and the second image, generating a third image byextracting the remaining portion of the first image excluding the secondimage and the kidney border image, and transmitting the second image,the kidney border image, and the third image to the image integrationunit.
 19. The apparatus according to claim 4, wherein theregion-of-interest extraction unit comprises: a wavelet frame unitacquiring a sub-band HL frame image, a sub-band LH frame image, and asub-band HH frame image by wavelet frame transform of the referenceorgan image and the ultrasound image; a feature point detection unitcomprising feature point determination units applying a feature pointdetermination threshold to the respective sub-band frame images on apixel-by-pixel basis and an overlapping pixel determination unitdetecting overlapping pixels by performing pixel-by-pixel multiplicationbetween the sub-band frame images with respect to pixels having a valuegreater than the feature point determination threshold, the featurepoint detection unit selecting overlapping pixels of the reference organimage acquired by the overlapping pixel determination unit as areference feature point and selecting overlapping pixels of theultrasound image acquired by the overlapping pixel determination unit asa matching feature point; a feature point pair generation unit comparingthe reference feature point with respective matching feature points inpatch images, which are formed around the respective matching featurepoints, to select patch images having a matching feature point with highsimilarity to the reference feature point and generating a pair offeature points between the reference organ image and each of theselected patch images; an outlier detection unit eliminating abnormalpairs of feature points among the generated pairs of feature points; ageometric parameter calculation unit calculating geometric parametersrequired for registration of the resultant pair of feature points, thegeometric parameters comprising translation, rotation, and scaling; andan organ matching unit acquiring a registered patch image using thegeometric parameters, calculating a cross-correlation coefficientbetween the reference organ image and the registered patch image,determining that a corresponding organ is located in a current patchimage in the ultrasound image if the cross-correlation coefficient isgreater than a predetermined threshold, extracting an organ image regionat a corresponding location, and transmitting the organ image region tothe image integration unit.